System and method for multi-modal audio mining of telephone conversations

ABSTRACT

A system and method for the automated monitoring of inmate telephone calls as well as multi-modal search, retrieval and playback capabilities for said calls. A general term for such capabilities is multi-modal audio mining. The invention is designed to provide an efficient means for organizations such as correctional facilities to identify and monitor the contents of telephone conversations and to provide evidence of possible inappropriate conduct and/or criminal activity of inmates by analyzing monitored telephone conversations for events, including, but not limited to, the addition of third parties, the discussion of particular topics, and the mention of certain entities.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/974,314, filed May 8, 2018, which is a continuation of U.S.application Ser. No. 15/412,992, filed Jan. 23, 2017, now U.S. Pat. No.10,120,919, issued Nov. 6, 2018, which is a continuation of U.S.application Ser. No. 14/281,323, filed May 19, 2014, now U.S. Pat. No.9,552,417, issued Jan. 24, 2017, which is a continuation of U.S.application Ser. No. 12/032,200, filed Feb. 15, 2008, now U.S. Pat. No.8,731,934, issued May 20, 2014, which claims the benefit of U.S.Provisional Application No. 60/901,342, filed Feb. 15, 2007, all ofwhich are incorporated by reference herein in their entireties.

BACKGROUND OF THE INVENTION a. Field of the Invention

This invention relates generally to telephony, and more particularly toa method and system for automatically monitoring and retrieving thecontent of telephone calls.

b. Background Art

There is a general need for systems that automatically monitor thecontents of telephone conversations. There is also a general need formethods that analyze the contents of monitored telephone conversationsfor a plurality of characteristics such as, for example, topics, phrases(e.g., the use of slang suggesting inappropriate behavior or criminalactivity), or evidence of the addition of third parties so that thecalls may later be retrieved on the basis of these characteristics.

Although most correctional facilities record all telephone calls made byinmates, it is believed that only a very small proportion of the callsis monitored by correctional officers. Many correctional facilitiesrecord 2,000 inmate telephone calls a day, such that monitoring alltelephone calls, or even a large fraction of all calls, on a regularbasis would require too much personnel and would be cost prohibitive.Inmates are aware of this and know that there is little chance that anyone particular conversation will be monitored. Thus, it is thought thatinmates may use telephone calls to engage in inappropriate conduct orcriminal behavior while incarcerated.

Research has shown that inmate telephone recordings often containevidence of inappropriate or illegal activity, such as terroristactivity, the intelligence value of which is not fully exploited bycorrectional facilities. This may result in the loss of potentiallyvaluable information that may help solve or prevent additional crimes.For example, it has been shown that inmates may use telephones toinappropriately contact witnesses, make drug deals, discuss gang relatedissues, and in general continue to exert criminal influence on the verycommunities in which they were convicted. Inmates may also discussissues related to non-criminal activities that are of interest tocorrectional officers, including suicidal thoughts or tendencies,medical or psychological problems, and political or religiousradicalization.

Patterns of inappropriate or criminal activity may not always be readilyapparent from a single telephone conversation. Rather, patterns may onlyappear when information from multiple conversations is combined, forexample from a single inmate or multiple inmates, from a singlecorrectional facility or multiple correctional facilities, or acombination thereof. Thus, even if all inmate telephone conversations atall correctional facilities were to be monitored, some patterns mightstill not be recognized unless the same individual monitors allconversations. Doing so would be physically impossible at mostcorrectional facilities.

From an investigative viewpoint, as well as from strategic and tacticalintelligence viewpoints, it is also desirable, whenever possible, tocombine data from multiple sources, such as inmate telephone systems,inmate management systems, and investigative case management systems.Each source of information may provide additional insight into patternsof inmate activity, and may also assist in establishing evidence ofinappropriate or criminal activity.

In certain cases it is imperative that correctional officials arenotified as soon as possible when an inmate discusses a specific topic.This is especially true for instances where threats of death, violence,destruction, or suicide are uttered during conversations. The sooner lawenforcement or intelligence officials become aware of these threats, thegreater the chances that they can prevent individuals from corning toharm.

It is also well established that it is of advantage to view data inmultiple formats, or modes. This is especially true in instances where aquery results in large amounts of data. For example, in addition totabular representations of data, data might also be viewed on a map, alink chart, or a timeline chart. Each mode of visualization can provideadditional insight into any patterns evident in the data that othermodes of visualization may not provide.

BRIEF SUMMARY OF THE INVENTION

It is therefore desirable to be able to automate the monitoring ofinmate telephone calls and provide correctional facilities with theability to search for and retrieve telephone calls that containpotential evidence of inappropriate conduct or illegal activity.

It is also desirable to be able to go directly to the point in theconversation where the inappropriate conduct or illegal activity wasdiscussed without having to listen to the entire conversation.

In addition, it is desirable to provide correctional officers with theability to search for and retrieve telephone calls in multiple modes,i.e. query from and display on multiple modes of visualization andanalytical techniques. Multi-modal queries and analytical techniquesassist in identifying patterns across calls and correctional facilities.

It is also desirable to combine data from multiple sources and/ormultiple correctional facilities to provide further insight intopatterns of inappropriate conduct or criminal activity and assist inestablishing evidence of inappropriate conduct or criminal activity thata single source of information may not reveal.

Also, it is desirable to provide a means with which correctionalofficials can instruct the monitoring system to immediately notify themwhen certain preset or user defined characteristics are detected in atelephone conversation.

The present invention provides a method and a system for the automatedmonitoring of inmate telephone conversations as well as multi-modalsearch, retrieval and playback capabilities for the monitored telephoneconversations. A general term for such capabilities is multi-modal audiomining. The invention is designed to provide an efficient means fororganizations such as correctional facilities to identify and monitorthe contents of telephone conversations and to provide evidence ofpossible inappropriate conduct or criminal activity of inmates ininstances where such activity indeed occurs.

Disclosed herein is a system for data-mining a monitored telephoneconversation. The system generally includes a speech recognitionprocessor configured to transcribe the monitored telephone conversationand associate a plurality of characteristics of the monitored telephoneconversation with a transcript of the monitored telephone conversation;an event detection processor configured to analyze the monitoredtelephone conversation to detect one or more events within the monitoredtelephone conversation; a tagging processor configured to associatemetadata information indicative of the detected one or more events withthe transcript of the monitored telephone conversation; and a multimediadata warehouse configured to store at least the transcript of themonitored telephone conversation and the metadata information andcharacteristics associated therewith as a data record for the monitoredtelephone conversation. The system may also include a data normalizationprocessor configured to atomize the data record for the monitoredtelephone conversation with respect to a plurality of data recordsstored within the multimedia data warehouse and a link creationprocessor configured to create one or more logical links between thedata record for the monitored telephone conversation and one or more ofthe plurality of data records stored within the multimedia datawarehouse. In addition, an optional synchronization processor mayassociate timing information for the detected one or more events with asound recording of at least a portion of the conversation or positioninformation (e.g., word or character counts) for the detected one ormore events with the transcript of the monitored telephoneconversation).

In some embodiments of the invention, the data record may include asound recording of at least a portion of the monitored telephoneconversation. Alternatively, a sound recording of at least a portion ofthe monitored telephone conversation may be stored in a sound recordingdatabase, and a reference thereto may be stored in the multimedia datawarehouse as part of the data record.

The speech recognition processor will typically utilize at least onedomain-specific language model to transcribe the monitored telephoneconversation, for example a language model that is domain-specific toinmate speech. It is also contemplated that the language model or modelsmay be domain specific to a plurality of regional dialects, a pluralityof ethnic groups, or a plurality of languages.

The event detection processor may include a three-way call detectionprocessor configured to analyze the plurality of characteristics for anindication of a third party being added to the monitored telephoneconversation.

The event detection processor typically also includes an entityidentification processor configured to analyze at least the transcriptof the monitored telephone conversation to identify one or more entitiesmentioned during the monitored telephone conversation. At least oneentity database may be coupled to the event detection processor. The atleast one entity database may include at least one domain-specificHidden Markov Model, such as a Hidden Markov Model domain-specific toinmate speech. The Hidden Markov Model or Models may also bedomain-specific to a plurality of regional dialects, a plurality ofethnic groups, or a plurality of language. Further, it is contemplatedthat the Hidden Markov Models may be configured to utilize trigrammaticword transitional probabilities, bigrammatic word transitionalprobabilities, or both.

The event detection processor may utilize any suitable algorithm toidentify one or more entities mentioned during the monitored telephoneconversation. For example, the entity identification processor mayutilize one or more Viterbi search or string matching algorithms tosearch the at least one entity database. Alternatively, the entityidentification processor may use one or more heuristic algorithms toheuristically identify one or more entities mentioned during themonitored telephone conversation.

The event detection processor may also include a topic detectionprocessor configured to analyze at least the transcript of the monitoredtelephone conversation to identify one or more topics discussed duringthe monitored telephone conversation. At least one topic database may becoupled to the topic detection processor. The topic detection processormay utilize one or more text clustering or phrase-matching algorithms tosearch the at least one topic database in order to identify one or moretopics discussed during the monitored telephone conversation.Optionally, the topic detection processor may be configured to performautomatic topic annotation of the conversation using at least onedomain-specific Hidden Markov Model and at least one domain-specifictopic language model stored in the at least one topic database. As withother domain-specific aspects of the invention, the Hidden Markov Modeland the topic language model used for topic detection may bedomain-specific to inmate speech or to one or more ethnic groups,regional dialects, languages, or other linguistic differences, and theHidden Markov Model may utilize either or both of trigrammatic andbigrammatic transitional word probabilities.

In some embodiments of the invention, at least one of a spatialinformation database and a telephone subscriber information database iscoupled to the event detection processor. The event detection processormay be configured to associate information retrieved from at least oneof the spatial information database and the telephone subscriberinformation database with at least the transcript of the monitoredtelephone conversation based upon the one or more events detected withinthe monitored telephone conversation.

A multi-modal query interface configured to accept user-specified searchcriteria may also be included. A search processor may be configured tosearch the multimedia data warehouse and retrieve one or more datarecords meeting the user-specified search criteria, and a multi-modaloutput interface may be configured to display the retrieved one or moredata records. In other embodiments of the invention, a monitoringprocessor is configured to monitor the multimedia data warehouse foraddition of a data record matching the user-specified search criteria. Anotification processor may then notify a user upon addition of the datarecord meeting the user-specified search criteria.

Also disclosed is a system for data-mining a monitored telephoneconversation, including: a multimedia data warehouse configured to storea plurality of data records, each of the data records associated with amonitored telephone conversation; a data transformation processorconfigured to receive data from at least one data source, the receiveddata including at least audio of a monitored telephone conversation andmetadata information regarding the monitored telephone conversation; adata augmentation processor configured to augment the received data withdata retrieved from one or more of a telephone subscriber database and aspatial information database; a speech recognition processor configuredto transcribe the monitored telephone conversation and associate aplurality of characteristics of the monitored telephone conversationwith a transcript of the monitored telephone conversation; and a datamapping processor configured to store a data record for the monitoredtelephone conversation to the multimedia data warehouse, the data recordincluding at least the transcript of the monitored telephoneconversation, the plurality of characteristics of the monitoredtelephone conversation, the metadata information regarding the monitoredtelephone conversation, and the retrieved data. The system optionallyincludes a data cleaning processor configured to clean the receiveddata, and, in some embodiments, to verify the suitability of thereceived data for further processing.

An optional data normalization processor is configured to atomize thedata record for the monitored telephone conversation with respect to themultimedia data warehouse, while an optional link creation processor isconfigured to create one or more logical links between the data recordfor the monitored telephone conversation and one or more data recordswithin the multimedia data warehouse. The data normalization processormay use a fuzzy logic algorithm, a matching algorithm (e.g., a soundexalgorithm), a natural language processing algorithm, or another suitablealgorithm to atomize the data record with respect to the multimedia datawarehouse.

The at least one data source will typically include an inmate telephonesystem (e.g., a system for monitoring and recording inmate telephonecalls, typically from a single correctional institution). The at leastone data source may also include an inmate management system and/or aninvestigative case management system. Further, the at least one datasource may originate from a single correctional facility or a pluralityof correctional facilities.

The invention also provides a method of data-mining a monitoredtelephone conversation including the following steps: transcribing amonitored telephone conversation; extracting a plurality ofcharacteristics of the monitored telephone conversation; associating theextracted plurality of characteristics of the monitored telephoneconversation with a transcript of the monitored telephone conversation;analyzing the monitored telephone conversation to detect one or moreevents within the monitored telephone conversation; associating metadatainformation indicative of the detected one or more events with thetranscript of the monitored telephone conversation; and storing at leastthe transcript of the monitored telephone conversation and thecharacteristics and metadata information associated therewith in amultimedia data warehouse as a data record for the monitored telephoneconversation. The method optionally further includes: atomizing the datarecord for the monitored telephone conversation with respect to aplurality of data records stored within the multimedia data warehouse;and creating one or more logical links between the data record for themonitored telephone conversation and one or more of the plurality ofdata records stored within the multimedia data warehouse. Atomicity maybe ensured by using any of a number of suitable algorithms, including,without limitation, fuzzy logic algorithms, matching algorithms (e.g.,soundex algorithms), and natural language processing algorithms. Thedetection of events may include detecting the addition of one or morethird parties to the conversation, the discussion of one or more topicsduring the conversation, and/or the mention of one or more entitiesduring the conversation.

In another aspect of the invention, a method of data-mining a monitoredtelephone conversation includes: receiving data from at least one datasource, the received data including at least audio of a monitoredtelephone conversation and metadata information regarding the monitoredtelephone conversation; retrieving data from one or more of a telephonesubscriber database and a spatial information database based upon themetadata information regarding the monitored telephone conversation;associating the retrieved data with the received data; transcribing themonitored telephone conversation; extracting a plurality ofcharacteristics of the monitored telephone conversation; associating theextracted plurality of characteristics of the monitored telephoneconversation with a transcript of the monitored telephone conversation;generating a data record for the monitored telephone conversation, thedata record including at least the transcript of the monitored telephoneconversation, the plurality of characteristics of the monitoredtelephone conversation, the metadata information regarding the monitoredtelephone conversation, and the retrieved data; and storing the datarecord for the monitored telephone conversation to a multimedia datawarehouse. The method may also include cleaning the received data.Cleaning the data may include one or more of the following steps:stripping leading blanks from the audio of the monitored telephoneconversation, stripping trailing blanks from the audio of the monitoredtelephone conversation, standardizing telephone numbers within themetadata information regarding the monitored telephone conversation,spell-checking names within the metadata information regarding themonitored telephone conversation, verifying suitability of the receiveddata for further processing, and any combinations thereof.

An advantage of the present invention is that it operates independentlyof any particular type of telephone system, such as VOIP or POTS.

Another advantage of the present invention is that it works with anyexternal data source that utilizes an open database architecturestandard.

Still another advantage of the present invention is that it allowscorrectional officers, intelligence analysts, and other officials toconcentrate their efforts on analysis rather than data collection andtransformation. The data is always available in the format that it isneeded and when it is needed.

A further advantage of the present invention is that it allowscorrectional officers to perform multiple modes of analysis andvisualization within a single system.

Another advantage of the present invention is that it permitscorrectional officers to search for and retrieve telephone conversationson the basis of topics discussed without having to know the exact wordsthat were uttered.

Yet another advantage of the present system is that it permits one to“jump” directly to the point in the telephone conversation where thesearch criteria were found without having to listen to the entiretelephone conversation.

Another advantage of the present invention is that the monitoring oftelephone conversations is fully automated, thus resulting inconsiderable manpower savings.

Still another advantage of the present invention is that it allowscorrectional officers to be automatically notified as soon as atelephone conversation meeting preset or user-defined criteria ismonitored, which may prevent a crime from occurring or a person fromcoming in harms way.

Yet a further advantage of the present invention is that it allows forthe analysis of patterns of telephone conversations across multiplecorrectional facilities.

The foregoing and other aspects, features, details, utilities andadvantages of the present invention will be apparent from reading thefollowing description and claims, and from reviewing the accompanyingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a data transfer system accordingto an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a data cleaning, processing andmapping system according to an embodiment of the present invention.

FIG. 3 is a block diagram illustrating a data augmentation process, suchas that shown in FIG. 2 .

FIG. 4 is a block diagram illustrating a speech recognition systemaccording to an embodiment of the present invention.

FIG. 5 is a block diagram illustrating a transcript and callcharacteristic processing method according to an embodiment of thepresent invention.

FIG. 6 is a block diagram illustrating a database agent and usernotification system according to an embodiment of the present invention.

FIG. 7 is a block diagram illustrating a generalized multi-modal querysystem according to an embodiment of the present invention.

FIG. 8 is a block diagram illustrating a generalized multi-modal queryretrieval system according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The following description illustrates the invention with reference to areal world domain, and in particular with respect to correctionalfacilities. However, it should be understood that the practice of theinvention is not limited to any particular domain or application.Rather, it is intended to be more generally applicable to any domainwhere there is a need for the automated monitoring and multi-modal queryand retrieval of recorded or live telephone conversations.

The present invention allows users, such as law enforcement personnel,correctional officers, intelligence analysts, and investigators, toautomatically monitor telephone conversations and search for evidence ofinappropriate conduct, criminal activity, or other topics of discussionthat should be brought to their attention. While prior art has typicallyfocused on simple key word searches of transcripts of recorded telephoneconversations, the present invention advantageously enables users toquery, retrieve, visualize and analyze the contents of monitoredtelephone conversations (also referred to herein as “telephoneconversations, “conversations,” or simply “calls”) in a plurality ofmodalities, as well as across a plurality of correctional facilities.Further, the present invention provides the ability to integrate otherdata known about persons, such as the inmates making the calls andcalled parties. Examples of such data include, without limitation,telephone subscriber identity, addresses, and type of service, recordsfrom inmate management systems, and data from investigative casemanagement systems.

FIG. 1 illustrates a block diagram of a data transfer system 100according to an embodiment of the present invention. Data transfersystem 100 may be software-implemented (e.g., a software programexecuted by one or more computer systems or processors),hardware-implemented (e.g., a series of instructions stored in one ormore solid-state devices), or a combination of both. It should also beunderstood that multiple instances of the data transfer system may besimultaneously executed on a single computer or on multiple computers.The term “processor” as used herein refers to a computer microprocessor,and/or a software program (e.g., a software module or separate program)that is designed to be executed by one or more computer microprocessorsrunning on one or more computer systems.

In FIG. 1 , a data transfer process 108 receives data from at least onedata source, such as external databases 102, 104, and 106, and forwardsthe received data to a data transformation process 110. One of ordinaryskill in the art will appreciate that data transfer process 108 and datatransformation process 110 may be implemented as separate processors orintegrated into a single processor. Data transformation process 110 mapsthe received data to a temporary database schema and stores it forsubsequent processing in a multimedia data warehouse holding area 112.

In some embodiments of the invention, the external databases include aninmate management system 102, an inmate telephone system 104, whichpreferably includes recorded, real-time, or near real-time audio ofmonitored inmate telephone conversations and information regarding themonitored telephone conversations, as well as a generic databasecontaining other data sources 106 that might include, for example, aninvestigative case management system. It is contemplated, however, thatthe invention may be practiced with only a single data source thatincludes at least audio of a monitored telephone conversation andinformation regarding the monitored telephone conversation, such as theidentity of the caller, the phone number called, the date and time ofthe call, and the duration of the call. For purposes of thisapplication, and as one of ordinary skill will understand, an inmatetelephone system is a telephone system that provides telephone servicesto the inmates for a corrections facility; typically, it containshardware and software that is housed in the facility, but it may also belocated at a remote site. Similarly, a inmate management system is acombination of hardware and software that permits a corrections facilityto process and monitor its inmates. An inmate management system istypically separate from an inmate telephone system, but they may beelectronically coupled to share information about inmates. As one ofordinary skill will appreciate, an investigative case management systemis a combination of hardware and software that permits a user (typicallya person who works in the corrections industry) to monitor the processand status of one or more individuals who have been processed at sometime in a corrections facility.

The external databases 102, 104, 106 may reside on a single computer ormultiple computers. Further, it is contemplated that a single databasemay be stored across multiple computers or storage devices. In addition,the external databases 102, 104, 106 may be provided by and processedfrom a plurality of correctional facilities. The data transfer process108 may include stored procedures that run on the external databases102, 104, 106, software programs that are executed on one or morecomputers and that access the external databases 102, 104, 106 remotely,or a combination of both.

In the multimedia data warehouse holding area 112, sound recordings (inwhole or in part) of monitored telephone conversations received from theinmate telephone system 104 may be stored internally as binary largeobjects (BLOBs). Alternatively, sound recordings of monitored telephoneconversations may be stored in an external file system, in which casethe multimedia data warehouse holding area 112 preferably includes areference to the external file system.

FIG. 2 is a block diagram illustrating a data cleaning, processing andmapping system 200, which may be software-implemented (e.g., a softwareprogram executed by one or more computer systems or processors),hardware-implemented (e.g., a series of instructions stored in one ormore solid-state devices), or a combination of both. It should also beunderstood that one or more instances of the data cleaning, processingand mapping system 200 may be executed simultaneously on one or morecomputers.

A data retrieval process 202 retrieves a record (e.g., the data receivedfrom databases 102, 104, 106) from the multimedia data warehouse holdingarea 112 and passes it to a data cleaning process 204. The data cleaningprocess 204 cleans the record (e.g., each element of the record) byperforming a plurality of functions including, but not limited to,stripping leading and trailing blanks or non alphanumeric charactersfrom strings; standardizing telephone numbers (e.g., the phone numbercalled, which may be included in the information regarding the monitoredtelephone conversation); ensuring that a minimum number of elements arepresent in address records so that these records may later be processedfor mapping; checking the spelling of first and last names (e.g., thefirst and last names of the inmate making the call, or the first andlast names of the called party, which may be included in the informationregarding the monitored telephone conversation); and flagging thoserecords where the quality of individual data elements is so inadequateas to render the entire record unusable and unsuitable for furtherprocessing, in which case the record may be discarded in someembodiments of the invention.

After successful cleaning, the data record may be passed on to the dataaugmentation process 208. Wherever possible, the data augmentationprocess 208 adds further information to the data record in order to makeit more useful for analytical purposes and to allow additionalcross-referencing with other records. Typically, the data augmentationprocess 208 will utilize the received data, such as the informationregarding the monitored telephone conversation, to retrieve data fromone or more databases. For example, in FIG. 2 the data augmentationprocess is coupled to a telephone subscriber database 206 and a spatialinformation database 214. Thus, the received data may be augmented byretrieving subscriber information for telephone numbers from subscriberdatabase 206 and spatial information, such as geographic coordinates foraddresses corresponding to the telephone numbers called, from spatialdata warehouse 214.

By augmenting the received data with the retrieved data, the ability ofthe present invention to query and display in multiple modalities (e.g.,query and display on the basis of inmate name, dialed phone number, nameof called party, address of called party, ZIP code of called party,police precinct of called party, and the like) is enhanced. For example,augmenting a land based telephone number of a called party withsubscriber information (e.g., name of subscriber and type of service)and address information allows the location of the telephone number tobe displayed on a map and queried based on location. The dataaugmentation process is illustrated in more detail in FIG. 3 , which isfurther discussed below.

Next the record may be passed to the data integration process 210. Thedata integration process performs a final check to ensure that eachelement of a record (e.g., the received data and the retrieved data) canreadily be integrated with a plurality of application specific datawarehouse schemas. It should be understood that the term “applicationspecific data warehouse” refers to data warehouses that are specializedin certain applications, such as a spatial data warehouse that storesdata used by geographic information systems. By using a plurality ofapplication specific data warehouses, the present invention is betterable to provide multiple modalities for query and visualization ofresults.

A data normalization and schema mapping process 212 maps each dataelement within the data record to the plurality of application specificdata warehouses and ensures that each entity (e.g., address, person,organization, telephone number) is only represented once within theplurality of application specific data warehouses (referred to herein asatomization and/or atomicity). Each record or element of a record maythen be inserted into a corresponding application specific datawarehouse, including links, references, and/or pointers to otherapplication specific data warehouses as necessary. Typical applicationspecific data warehouses include a multimedia data warehouse 216 and aspatial data warehouse 214. The collective schema of data warehouses isreferred to herein as a “multimedia data warehouse.”

As mentioned above, one of the functions of the data normalization andschema mapping process 212 is to prevent duplication of data. That is,the data normalization and schema mapping process 212 ensures atomicityof the data record for a monitored telephone conversation relative tothe multimedia data warehouse. From an analytical perspective, dataduplication can result in missed linkages or patterns. Thus, it isdesirable for the data normalization and schema mapping process 212 tomaintain master tables within the plurality of application specific datawarehouses (e.g., within the multimedia data warehouse). In doing so,the data normalization and schema mapping process 212 may utilize anumber of methods including, without limitation, fuzzy logic, soundexand other matches, and natural language processing techniques. Forexample, the data normalization and schema mapping process 212 maymaintain a master table of called parties. If the called party in aparticular monitored telephone conversation is unique, an entry is addedto the master table of called parties. If, however, the called party isalready within the master table, no entry is added to the master table,though a link may be created as further described below.

As described above, where an element of a data record consists of binarydata, such as a sound recording (in whole or in part) of the monitoredtelephone conversation, the data element may be inserted directly as abinary large object (BLOB) within the appropriate application specificdata warehouse within the multimedia data warehouse. Alternatively, thedata element may be stored on an external file system, with a referenceto the data element within the external file system stored in themultimedia data warehouse.

FIG. 3 is a block diagram illustrating the data augmentation process 208depicted in FIG. 2 in further detail. FIG. 3 illustrates a number ofadditional aspects of the data augmentation process 208 that may beincorporated into various embodiments of the present invention,depending in part upon the type of data that is being processed.

One aspect of the data augmentation process 208 involves augmentation oftelephone numbers within the received data, such as the telephone numberof the called party. To this end, a subscriber lookup process 306 may becoupled to an external subscriber database 206 that includes informationsuch as the subscriber's name, the subscriber's address, the type ofservice (e.g., cell, POTS, VOIP), the type of subscriber (e.g., private,business, organization) and the name of the telephone company providingthe service. The subscriber lookup process may access the subscriberdatabase 206 and retrieve any or all of this information and associateit with the data record for the monitored telephone conversation.

Another aspect of the data augmentation process 208 involvesaugmentation of addresses within the data record. Thus, an addressmatching and standardization process 304, which may be coupled to thespatial data warehouse 214, may perform a number of functions to augmentaddresses with geographic features. This advantageously facilitatesvisualization and query of data from a geographic information system(GIS) interface. The processor preferably references at least oneaddress layer stored within the spatial data warehouse 214. The addresslayer may be line based, where the geographic coordinates of an addressare derived via interpolation along a street segment, point based, wherethe geographic coordinates of an address are derived directly based uponthe polygon centroids of land parcels or buildings, or a combination ofboth line based and point based. Typical functions performed by theaddress matching and standardization process 304 include, but are notlimited to, address matching, spatial overlays, and addressstandardization. Spatial overlays add information that is not present inthe original address record such as, but not limited to, census tractnumbers, census block numbers, county information, and police zones orpatrol sectors. This additional information can then be used forstrategic analyses of calling patterns. The spatial data warehousepreferably includes at least one spatial index of the spatial featuresextracted in the address matching and standardization process 304.

Also shown in FIG. 3 as an aspect of the data augmentation process 208is a link construction process 302. The link construction process 302creates logical links between entities such as addresses, persons,organizations, places, and telephone numbers so that data may bevisualized and queried from a link charting interface. The linkconstruction process 302 adds intersection entries between atomicentities (e.g., links between the data record for the monitoredtelephone conversation and one or more data records or master tableentries within the multimedia data warehouse) that describe each linkthat has been established within the data. Examples of such intersectionentries include, but are not limited to, inmates that called a specifictelephone number, subscribers of telephone numbers, home addresses ofinmates, relatives of inmates, and an address linked to a telephonenumber. As described above in connection with data normalization andschema mapping processor 212, it is desirable to verify the existence ofatomic entities within the multimedia data warehouse 216 and the spatialdata warehouse 214 (e.g., within the multimedia data warehouse) beforeconstructing links.

FIG. 4 is a block diagram illustrating a speech recognition system 400according to an embodiment of the present invention. The speechrecognition system of FIG. 4 may be software-implemented (e.g., asoftware program executed by one or more computer systems or speechrecognition processors), hardware-implemented (e.g., as a series ofinstructions stored in one or more solid-state devices), or acombination of both. It should also be understood that multipleinstances of the speech recognition system of FIG. 4 . may besimultaneously executed on a single computer or on multiple computers.

In FIG. 4 the speech recognition process 404 processes previouslyrecorded telephone conversations 406 or ongoing telephone conversations402, which are referred to interchangeably herein as “monitoredtelephone conversations”, “telephone conversations”, “conversations”, orsimply “calls”. Previously recorded telephone conversations may beobtained through a call retrieval process 414 that retrieves calls thathave not yet been speech recognized from the multimedia data warehouse216 or the external file system in which the sound recordings are storedbased upon references within the multimedia data warehouse 216.

Speech recognition process 404 performs a number of functions, of whichone is converting the spoken audio to text (transcription). Preferably,the entire conversation is transcribed, but it is contemplated that onlyportions of the conversation may be transcribed as well. In doing so,the speech recognition process utilizes at least one language model1408. When transcribing speech to text, it is desirable to ensure thatthe language model used is domain specific. A “domain specific” languagemodel is a language model that accurately reflects the linguisticnuances of the participants of a telephone conversation, for example alanguage model that is domain specific to inmate speech or telephony.Preferably, therefore, at least one domain specific language model isused.

In some embodiments of the invention, it is contemplated that multipledomain specific language models may be used, which may be trained for aplurality of ethnic groups, a plurality of regional dialects, or otherlanguage differences. Using multiple domain specific language models hasbeen shown to significantly improve speech recognition and transcriptionaccuracy. It is also contemplated that, in instances where foreignlanguages are spoken, multiple domain specific language models trainedfor a plurality of foreign languages may be used. Further, a translationprocessor may be utilized to translate the transcript of the monitoredtelephone conversation from a first language to a second language (e.g.,to translate a conversation from Spanish into English).

In addition to converting spoken audio into text, the speech recognitionprocess extracts a number of verbal and non-verbal characteristics fromthe telephone conversation. These include, but are not limited to,speaker turns (e.g., as determined by voice recognition of the speakersin the telephone conversation); gaps in audio; dial, pulse and ringtones; verbal cues (e.g., mentions of telephone numbers, mentions ofpeople or organizations to call, or telephone salutations such as“hello,” “hi,” and the like); speech and phraseology patterns; andtiming information that includes the beginning and end times (e.g., asmeasured in either seconds or milliseconds from the beginning of thetelephone conversation) of utterances, audio gaps, dial, pulse and ringtones, and speaker turns. The characteristics are preferably associatedwith the transcript of the monitored telephone conversation.

Once the speech recognition process has completed processing thetelephone conversation, it outputs the results to a file. One suitableformat for the output file is an XML file 412 the output file is thenprocessed by an output processor 410 that extracts each component fromthe XML file and inserts it as a call record into the multimedia datawarehouse 216, for example as a binary large object (BLOB). That is, asillustrated in FIG. 4 , the multimedia data warehouse 216 stores atleast the transcript of the monitored telephone conversation and theassociated characteristics of the monitored telephone conversation. Thetelephone conversation is now ready for further processing.

FIG. 5 is a block diagram illustrating a transcript and callcharacteristic processing system 500 according to an embodiment of thepresent invention. The transcript and call characteristic processingsystem 500 data-mines the monitored telephone conversation to detectevents, such as three-way calls, entities mentioned, and topicsdiscussed, such that the data record for the monitored telephoneconversation may be updated to include information indicative of theevents detected, which in turn improves the efficacy of queries of themultimedia data warehouse. It should be understood that multipleinstances of the transcript and call characteristic processing system ofFIG. 5 may be simultaneously executed on a single computer or onmultiple computers.

As shown in FIG. 5 , a call retrieval process 502 retrieves a telephoneconversation that has been previously processed by the speechrecognition system 400 from the multimedia data warehouse 216, includingat least the call transcript and any characteristics extracted by thespeech recognition system 400. Of course, the call retrieval process 502may also retrieve any other data that is related to or associated withthe data record that may be of use in the transcript and callcharacteristic processing system 500. Once the monitored telephoneconversation and associated data have been retrieved, an event detectionprocess, several aspects of which will be described in further detailbelow, analyzes the monitored telephone conversation to detect one ormore events therein. The term “events,” as used herein, includes, but isnot limited to, the addition of one or more third parties to themonitored telephone conversation, the mention of one or more entities(e.g., phone numbers; individual, business, organization, or place namesor abbreviations; addresses or locations; dates; times; monetaryamounts; slang phrases), and the discussion of one or more topics (e.g.,terrorism; suicide; drugs; violent crime).

One aspect of the event detection process detects three-way calls—thatis, the addition of one or more third parties to the monitored telephoneconversation. Thus, a three way call detection process 504 may analyzethe monitored telephone conversation (e.g., the transcript and theplurality of characteristics of the monitored telephone associatedtherewith) for evidence of one or more additions of third parties withinthe telephone conversation transcript and associated callcharacteristics. Upon successful detection of a three way call,information regarding the addition of a third party may be generated andadded to the plurality of call characteristics extracted in FIG. 5 .Also added to the characteristics in some embodiments of the inventionmay be a score that indicates the degree of confidence that a three waycall has been detected. This score can then later be used to restrictrecords returned as a result of queries to a minimum threshold ofconfidence and thus limit the number of false positive results. Onesuitable system and method for three way call detection is described inU.S. Pat. No. 8,542,802, though it is contemplated that other three waycall detection methods and systems may be employed without departingfrom the spirit and scope of the present invention.

Another aspect of the event detection process is the detection of one ormore entities mentioned during the monitored telephone conversation, aswell as, in some embodiments of the invention, the point during themonitored telephone conversation at which the entity was mentioned.Accordingly, upon completion of three way call detection, an entityextraction process 506 may search the transcript and extract entitiestherefrom. The term “entities” includes, but is not limited to, personnames, names and abbreviations of organizations, addresses, place names,telephone numbers, dates, times and monetary amounts.

In some embodiments of the invention, the entity extraction processutilizes at least one domain specific Hidden Markov Model (HMM) that isstored in an external entity database 514 and one or more suitablesearch algorithms including, but not limited to, Viterbi searches, tomaximize the likelihood that each utterance in the transcript does ordoes not represent all or part of an entity, and, if it does representall or part of an entity, what the nature of the entity is (e.g.,distinguishing a phone number from a monetary amount). A variety ofother techniques including, but not limited to, string matches of knownentities and heuristic methods to determine whether a sequence ofnumbers uttered during a telephone conversation represent a telephonenumber, may also be used as part of the entity extraction process.

If the entity extraction process determines that a telephone number hasbeen uttered in the telephone conversation, the telephone number may belooked up in the subscriber database 206 to verify whether it is indeeda registered telephone number. If it is a registered telephone number,subscriber information may be added as in the subscriber lookup process306 depicted in FIG. 3 and described above. Alternatively, the telephonenumber may simply be extracted from the monitored telephone conversationand associated therewith. Similarly, if the entity extraction processdetermines that an address or location has been uttered, the address orlocation may be looked up in a spatial information database, for exampleas described above in connection with the address matching andstandardization process 304.

It is desirable to ensure that the entity extraction process utilizes atleast one domain specific Hidden Markov Model that accurately reflectsthe linguistic nuances of the participants of a telephone conversation,for example, a Hidden Markov Model that is domain specific to inmatespeech or telephony. Preferably, therefore, at least one domain specificHidden Markov Model is used. In some embodiments of the invention, it iscontemplated that multiple Hidden Markov Models may be used, which maybe trained for a plurality of ethnic groups, a plurality of regionaldialects, or other language differences. It is also contemplated that,in instances where foreign languages are spoken, multiple domainspecific Hidden Markov Models trained for a plurality of foreignlanguages may be used. Using multiple domain specific Hidden MarkovModels has been shown to significantly improve recognition accuracywithin entity extraction processes and to minimize false positiveidentification of entities. It is also contemplated that in someembodiments of the present invention the plurality of Hidden MarkovModels may incorporate trigrammatic transitional word probabilities inaddition to bigrammatic transitional word probabilities.

Upon completion of the entity extraction process 506, a topic detectionprocess 508 may determine one or more topics discussed during thetelephone conversation and the point or points during the telephoneconversation the topics were discussed. In the preferred embodiment ofthe invention, the topic detection process 508 matches phrases containedin an external topic database, such as slang database 516. One advantageof using a slang database is that it provides a repository of slangutterances that may be made during a telephone conversation including,but not limited to, prison slang, gang slang, radical religious slang,terrorism related slang, slang for weapons, violence, and threats ofviolence, and drug related slang. Preferably, each slang term within theslang database is associated with both a major topic and a minor topic,though it is within the spirit and scope of the invention to associateany number of topics with a particular slang term. For instance, for theslang term “blow” a major topic might be drugs and a minor topic mightbe drug dealing. By detecting topics discussed during a monitoredtelephone conversation by analyzing at least the transcript of themonitored telephone conversation, a user, such as a correctionalofficer, may query the multimedia data warehouse for telephoneconversations involving discussions of, for example, drugs in generaland drug dealing in particular without having to know the exact phraseor phrases used during the conversation. Of course, additional slangcategories may also be incorporated within the topic database.

Other topic detection techniques are also regarded as within the spiritand scope of the present invention. Such techniques include, but are notlimited to: text clustering techniques; and at least one domain specificHidden Markov Model combined with at least one domain specific topiclanguage model that perform automated topic annotation of inmatetelephone conversations. In some embodiments of the invention, it iscontemplated that multiple domain specific Hidden Markov and topiclanguage models may be used, which may be trained for a plurality ofethnic groups, a plurality of regional dialects, or other languagedifferences. It is also contemplated that, in instances where foreignlanguages are spoken, multiple domain specific Hidden Markov and topiclanguage models trained for a plurality of foreign languages may beused. Using multiple domain specific Hidden Markov and topic languagemodels has been shown to significantly improve recognition accuracywithin topic segmentation and annotation processes and to minimize falsepositive annotation of topics. It is also contemplated that in someembodiments of the present invention the plurality of Hidden Markov andtopic language models may incorporate trigrammatic transitional wordprobabilities in addition to bigrammatic transitional wordprobabilities.

Once the topic detection process has been completed, a tagging process510 may tag the monitored telephone conversation with informationindicative of each event (e.g., each three way call, entity, slangphrase, and topic) of the telephone conversation being processed thatwas detected by the event detection process. The tags may then be addedto the multimedia data warehouse 216 as part of the call record, forexample as call record metadata (which is distinct from the underlyingactual data such as the sound recording of the telephone conversationand the transcript thereof).

The tagging process 510 then passes control to the synchronizationprocess 512. The synchronization process identifies timing informationfor the detected events (e.g., the likely beginning and, whereapplicable, end of each detected three way call, entity mention, phraseand topic discussion). For example, the synchronization process 512 mayadd one or, where applicable, two time stamps or other timinginformation to a sound recording of the monitored telephoneconversation, where the timing information identifies the number ofseconds or, in some embodiments, milliseconds from the beginning of thecall at which a three way call, utterance of an entity, or discussion ofa topic began and/or ended. The synchronization process 512 may also addposition information about how many words or characters into thetranscript a three way call, utterance of an entity, or discussion of atopic began and/or ended. Adding timing and/or position informationadvantageously may permit a user reviewing the monitored telephoneconversation to “jump” directly to the point in the conversation wherethe three-way call occurred, where a particular entity was mentioned, orwhere a particular topic was discussed, thereby avoiding the need toreview the entire conversation.

The multimedia data warehouse 216 may also be updated to include atleast one index of three way calls, topics and entities and thecharacteristics associated with them.

FIG. 6 is a block diagram illustrating a database agent and usernotification system 600 according to an embodiment of the presentinvention that may be used to create a database agent that continuouslymonitors the plurality of application specific data warehouses (e.g.,the multimedia data warehouse) for the addition of monitored telephoneconversations that involve a plurality of possible characteristics orthat meet preset or user-defined search criteria including, but notlimited to: specific inmates, called telephone numbers, discussedtopics, slang used, entities mentioned, three way calls, and anycombination of the preceding characteristics. It is also contemplatedthat the search criteria may be selected to permit the automatedmonitoring of patterns of activity, including, but not limited to:multiple calls from a plurality of inmates to the same telephone numberwhere drug dealing is a topic; calls from inmates where gang relatedtopics are detected where the called telephone number is located in ageographic area known to be frequented by gang members; and anytelephone conversation where a link is established between two entitiesthat was previously unknown, such as a three way call from an inmate toa telephone number that belongs to a known drug dealer.

As shown in FIG. 6 , a user 602, such as a corrections officer,investigator, or similar person, uses a software based agent wizard 604to create a software based database agent that is typically storedwithin the multimedia data warehouse 216, though the agent could bestored anywhere within the multimedia data warehouse. The database agentpreferably includes contact details for the user and at least one methodby which the user prefers to be notified in the event that the databaseagent detects the addition one or more call records that match thesearch criteria of the database agent.

An agent monitor 608, which is typically a software program, but may behardware-implemented without departing from the spirit and scope of thepresent invention, picks up the newly created database agent andcontinuously monitors the multimedia data warehouse (e.g., the spatialdata warehouse 214 and the multimedia data warehouse 216) for any newrecords or patterns that match the search criteria defined in thedatabase agent. Some embodiments of the present invention may deploymultiple agent monitors simultaneously on one or more computers. It isalso contemplated that a plurality of specialized agent monitors may beexecuted simultaneously including, but not limited to: applicationspecific agent monitors, agent monitors that specialize in patterndetection, and agent monitors that specialize in less complex queries.It should also be understood that, in addition to executing within themultimedia data warehouse as a stored procedure, the agent monitor mayalso be stored and executed externally as a software process.

Upon detecting the addition of a call record, pattern, or other criteriathat match the search criteria of a database agent operating on orwithin the multimedia data warehouse, the agent monitor 608 forwards thecall record, pattern or other criteria to an agent report generator 606.The agent report generator constructs a report that includes a pluralityof aspects of the call record, pattern, or other criteria. A report mayinclude, but is not limited to, call transcripts, subscriberinformation, inmate details, a detailed description of the identifiedpattern, and a listing of phrases that matched the database agentcriteria.

The report is then forwarded to a user notification manager 610. Theuser notification manager notifies the user that created the databaseagent via the at least one method by which the user prefers to benotified and, optionally, may also forward a copy of the reportgenerated by the agent report generator 606. Notification methodsinclude, but are not limited to: email, pager, voice mail or databasealerts that are triggered the next time the user logs in to the queryand retrieval systems, such as illustrated in FIG. 7 and FIG. 8 .Notification may also be provided by automated voice mail, in which casea text to speech processor that reads the report to voice mail may beincorporated in database agent and user notification system 600.

FIG. 7 is a block diagram illustrating a generalized multi-modal querysystem 700 according to an embodiment of the present invention that maybe used to retrieve records from the multimedia data warehouse (e.g.,the multimedia data warehouse 216 and the spatial data warehouse 214). Auser 602, such as a correctional officer, investigator, or similarperson, accesses a multi-modal query interface 702, which in turnconsists of a plurality of interfaces, each of which is application, ormode specific. In some embodiments of the invention, the mode specificinterfaces consist of a map interface 704, a link chart interface 706,and an ad hoc interface 708. Query criteria may be selected using one ormore of the mode-specific interfaces. For example, a user may want tosearch the multimedia data warehouse for all calls to a particulargeographic region, involving gang related topics, where a link existsbetween a number of certain inmates and at least one telephone numberfor which the subscriber address lies within the selected geographicregion.

Upon completion of the selection of query criteria, the criteria areforwarded to a query generator 710. The query generator translates themulti-modal query into, for example, SQL code that can then be appliedto the multimedia data warehouse 216, the spatial data warehouse 214,and any other application specific data warehouses within the multimediadata warehouse. After the SQL code has been constructed, the querygenerator 710 executes the query.

FIG. 8 is a block diagram illustrating a generalized multi-modal queryretrieval system 800 according to an embodiment of the presentinvention. Upon execution of the SQL code generated by the querygenerator 710 shown in FIG. 7 , a query retrieval manager 804 retrievesall data associated with the query and distributes the results to aplurality of modes of visualization within the visualization interface802. In the embodiment of the present invention illustrated in FIG. 8 ,the visualization interface consists of: a topic centric display whichdisplays the retrieved data arranged by topic; a toll call analysisdisplay which automatically performs a toll call analysis of telephonenumbers associated with the retrieved data; an inmate centric displaywhich arranges the retrieved data by inmate; a timeline display whichautomatically arranges all calls and patterns retrieved temporally; alink chart display which automatically creates a link chart showing theconnections between the entities, topics, phrases and other callcharacteristics retrieved by the query; and a map display which maps theaddresses associated with the data retrieved by the query.

It should be contemplated that other embodiments of the invention mayinclude other types or combinations of analytical display and modes ofvisualization without limitation. Other embodiments of the invention mayalso utilize additional external application servers in instances whereweb-based services are used. These include, but without limitation, ageographic information system application server and a link chartingapplication server. For modes of visualization that permit the playbackof retrieved recorded telephone conversation any characteristicsidentified by any of the previous processes may be visually identifiedwithin the transcript. For example, the point in a transcript where athree way call was detected may be highlighted and, optionally, where itwas terminated. In some embodiments of the invention time stampsgenerated by the synchronization process 512 illustrated in FIG. 5 maybe used to navigate (or “jump”) directly to the point in theconversation where the topic, entity, phrase or other detected callcharacteristic occurred, thereby advantageously permitting correctionsofficials to review only the portion of the conversation that is ofparticular interest. The system 800 may also include an output device orfacility capable of providing synchronized playback and visualization,respectively, of the sound recording and transcript of a selectedmonitored telephone conversation. For example, as the sound recording ofthe call is played back, a moving highlight may track through thetranscript.

Although only a few illustrative embodiments of this invention have beendescribed above with a certain degree of particularity, those skilled inthe art could make numerous alterations to the disclosed embodimentswithout departing from the spirit or scope of this invention. Forexample, one of ordinary skill in the art will appreciate that any ofthe systems disclosed herein (e.g., data transfer system 100, speechrecognition system 400) may be software-implemented (e.g., a softwareprogram executed by one or more computer systems or processors),hardware-implemented (e.g., a series of instructions stored in one ormore solid-state devices), or a combination of both, and that multipleinstances of such systems may be simultaneously executed on a singlecomputer or on multiple computers. In addition, it is contemplated thatthe various processors described herein may be integrated into one ormore processors incorporated into one or more computers withoutdeparting from the spirit and scope of the present invention. Further,it is contemplated that the present invention may be practiced inreal-time, near real-time, or on pre-recorded telephone conversations.

It is intended that all matter contained in the above description orshown in the accompanying drawings shall be interpreted as illustrativeonly and not limiting. Changes in detail or structure may be madewithout departing from the spirit of the invention as defined in theappended claims.

What is claimed is:
 1. A system for multi-modal audio mining oftelephone data, the system comprising: a communication database thatstores communications involving inmates of a correctional facility; andone or more processors configured to analyze a first communication fromthe communication database, the analyzing including: transcribing thefirst communication using speech recognition processing to generate atranscript of the first communication that includes non-verbalcharacteristics of the first communication; detecting an event withinthe first communication using audio analysis, the event including adetected topic of conversation with the first communication; comparingthe detected topic of conversation to a list of restricted topics ofconversation; identifying a rules violation based on the comparing;searching the communication database for a previously-recordedcommunication that includes the detected topic of conversation; andidentifying the previously-recorded communication as being related tothe first communication based on the searching and a similarity of thenon-verbal characteristics of the first communication to previousnon-verbal characteristics of the previously-recorded communication,wherein the non-verbal characteristics of the first communication andthe previous non-verbal characteristics of the previously-recordedcommunication include gaps in audio, dial, pulse and ring tones, verbalcues, speech patterns, and timing information.
 2. The system of 1,wherein the detecting of the event includes identifying a participant inthe first communication using speaker identification.
 3. The system ofclaim 2, wherein the identifying includes: isolating a speech sample ofa single speaker; generating a voiceprint based on the isolated speechsample; and comparing the generated voiceprint to previously-storedvoiceprints.
 4. The system of claim 1, wherein the detecting of theevent includes detecting a 3-way call based on tonal indicators in thefirst communication.
 5. The system of claim 1, wherein the one or moreprocessors are further configured to analyze the transcript formentioned entities, includeing names of individuals, groups, locations,and businesses.
 6. The system of claim 1, further comprising a topicdatabase that includes one or more keywords and/or keyphrases associatedwith different topics of interest.
 7. The system of claim 6, wherein theone or more processors are further configured to detect a topic ofinterest within the first communication using text clustering of phrasematching.
 8. A method for multi-modal audio mining of telephone data,comprising: storing recorded telephone calls involving inmates of acorrectional facility; retrieving a telephone call from among therecorded telephone calls for analysis; transcribing the retrievedtelephone call using speech recognition processing to generate atranscript of the retrieved telephone call that includes non-verbalcharacteristics of the retrieved telephone call; analyzing the audio ofthe retrieved telephone call; detecting an event within the retrievedtelephone call based on the analyzing, the event including a detectedtopic of conversation within the telephone call; comparing the detectedtopic of conversation to a list of restricted topics of conversation;identifying a rules violation based on the comparing; searching acommunication database for a previously-recorded communication thatincludes the detected topic of conversation; and identifying thepreviously-recorded communication as being related to the retrievedtelephone call based on the searching and a similarity of the non-verbalcharacteristics of the retrieved telephone call to previous non-verbalcharacteristics of the previously-recorded communication, wherein thenon-verbal characteristics of the telephone call and the previousnon-verbal charecteristics of the previously-recorded communicationinclude gaps in audio, dial, pulse and ring tones, verbal cues, speechpatterns, and timing information.
 9. The method of claim 8, wherein thedetecting of the event further includes identifying a participant in theretrieved telephone call using speaker identification.
 10. The method ofclaim 9, wherein the identifying includes: isolating a speech sample ofa single speaker; generating a voiceprint based on the isolated speechsample; and comparing the generated voiceprint to previously-storedvoiceprints.
 11. The method of claim 8, wherein the detecting of theevent includes detecting a 3-way call based on tonal indicators in theretrieved telephone call.
 12. The method of claim 8, further comprising:identifying keywords and/or keyphrases in the transcribed telephonecall; and comparing the identified keywords and/or keyphrases to storedkeywords and/or keyphrases in a topic database; and identifying a topicbased on the comparing.
 13. The method of claim 8, further comprising:identifying entities mentioned in the retrieved telephone call, theentities including names of individuals, groups, locations, andbusinesses.
 14. The method of claim 8, further comprising detecting atopic of interest by performing at least one of text clustering andphrase matching.